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ABSTRACT 
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work through an ill-structured problem, represented hierarchically and 
heterarchically in a Web-based format. A Web-based tracking program was 
deployed to track students' journeys through the hypermedia case designs. 
Students were observed while interacting with the problem and were interviewed 
after submitting their case solutions. Results from the tracking program, 
observations, case solutions, and interview questions address case design 
issues, problem-solving issues, and group processes. Findings reveal that 
heterarchical case designs potentially increase student collaboration in a 
Web-based hypermedia design of an ill-structured problem and engage students 
in thinking critically about case content. (Contains 3 tables, 2 figures, and 
34 references.) (Author/SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



Assessing Complex Problem-Solving Skills and Knowledge Assembly Using Web-Based 
Hypermedia Design 



r~~ 

r- 

\o 

VO 

m- 



O 



W 



Abstract 

This research project studied the effects of hierarchical versus heterarchical hypermedia 
structures of Web-based case representations on complex problem-solving skills and knowledge 
assembly in problem-centered learning environments in order to develop a system or model that 
informs the design of Web-based cases for ill-structured problems across multiple disciplines. 
Two groups of students were assigned to work through an ill-structured problem, represented 
hierarchically and heterarchically in a Web-based format. A Web-based tracking program was 
deployed to track students’ journeys through the hypermedia case designs. Students were 
observed while interacting with the problem and were interviewed after submitting their case 
solutions. Results from the tracking program, observations, case solutions, and interview 
questions will address case design issues, problem-solving issues, and group processes. 
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Assessing Complex Problem-Solving Skills and Knowledge Assembly Using Web-Based 
Hypermedia Design 

Research Problem 

Several research projects and studies (Booth-Sweeney, 2001; Herreid & Schiller, 2001; 
Siegel, et al., Spiro, 2000; Rogers & Erickson, 1998; Gerdes, 1998; Sutyak, et al., 1998; 
Fitzgerald & Semrau, 1996; Jacobson, et al., 1996) have investigated the use of cases or 
problems in instruction particularly in relation to case structure (e.g. linear versus hypertext, 
narrative versus conceptual) and problem complexity (e.g. well-defined versus ill-structured), 
and the impact of such variables on advanced knowledge acquisition. However to date, there is 
no explicit framework or instructional model to guide the design of Web-based hypermedia 
cases, particularly for ill-structured problems or learning tasks that require students to engage in 
complex problem-solving and knowledge transfer. In addition, Grissom and Koschmann (1995) 
contend that cases or problems that are used as a stimulus for authentic activity are hard to come 
by in disciplines other than medicine, business, and law, and suggest that hypertext/hypermedia 
may be a more efficient and effective medium to produce cases for learning environments that 
are problem-centered. 

This research project studied the effects of different hypermedia structures of Web-based 
case representations on complex problem-solving skills and knowledge assembly in problem- 
centered learning environments in order to develop a system or model that informs the design of 
Web-based cases for ill-structured problems across multiple disciplines. Ill-structured problems 
are the kinds of problems or tasks that are encountered in everyday practice requiring the 
integration of several content domains and possessing multiple solutions or solution paths 
(Jonassen, 1997). They are problems that are “situated in and emergent from a specific context” 
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(Jonassen, 1997, p. 68) and “lacking solutions that are indisputably correct” (Kagan, 1993, 
p.715). Therefore they are arguably most appropriate for engaging students in authentic activity 
and assessing complex problem-solving skills and knowledge transfer. 

Theoretical Framework 

Duffy and Cunningham (1996) identified five strategies for using problems in instruction: 
(1) the problem as an example; (2) the problem as an integrator or test; (3) the problem as a 
guide; (4) the problem as a vehicle for process; and (5) the problem as a stimulus for authentic 
activity. This research project focused on their strategy of using the problem as a stimulus for 
authentic activity. Authentic activity is most simply defined as the ordinary practices of a culture, 
namely, coherent, meaningful, and purposeful activities (Brown, Collins, & Duguid, 1989). More 
specifically, an authentic learning task for students should have the following characteristics. It 
should: (1) cue the leaner to the desired solutions) in order to promote “free exploration” or self- 
directed inquiry; (2) allow multiple solutions or solution paths; (3) have no explicit means for 
determining appropriate action; (4) be perceived as real or consequential by the learner to 
promote ownership of the problem; and (5) possess multiple criteria for evaluating solutions. 

Problems or learning tasks that possess the above characteristics are known as ill-defined 
or ill-structured problems (Jonassen, 1997, 2000). In mathematics education, these learning tasks 
are sometimes referred to as “thought-revealing” or “model-eliciting” problems (Lesh, et al., 
2000). In this genre of problems, the onus is placed on the student to create a model to capture in 
meaningful ways the complexity presented in the learning task. The central question for the task 
designer becomes, “Does the task put students in a situation where they recognize the need to 
develop a model for interpreting the givens, goals, and possible solution processes in a complex. 
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problem-solving situation? Or, does it ask them to produce only an answer to a question that was 
formulated by others?” (Lesh, et al., 2000, p. 609). 

Problem-based learning (PBL) is one of the most extensive and elaborate instructional 
models in designing authentic learning tasks (Dabbagh, et al., 2000). PBL is defined as “the 
learning that results from the process of working toward the understanding or resolution of a 
problem” (Barrows & Tamblyn, 1980, p. 18). The learner is expected to actively participate in 
the learning process and construct his/her own knowledge through direct experience and 
interaction with the situation, that is, knowing-in-action (Schon, 1987). As Stepien and Gallagher 
(1993) state: “Problem-based learning is apprenticeship for real-life problem-solving” (p. 26). 

PBL problems are typically ill-structured and complex, revealing the underlying principles and 
concepts of a knowledge domain through descriptions of real life events and experiences, and 
serving as the stimulus and focus for student activity (Siegel et al., 2000; Norman & Schmidt, 
1992; Stepien & Gallagher, 1993). PBL radically alters the role of the teacher from instructor to 
facilitator and as time goes on, learners become increasingly self-directed in the learning process 
(Shoflher & Dalton, 1997) by taking ownership of the problem and "assuming the roles of 
scientists, historians, doctors, or others who have a real stake in the proposed problem (Stepien & 
Gallagher, 1993).” 

Hypermedia Support for PBL 

The use of hypertext/hypermedia to represent PBL cases has been a primary interest to 
researchers of PBL due to its inherent non-linear interlinking ability in representing a complex 
and interconnected body of information (Hemstreet, 1997; Jacobson & Spiro, 1 995). According 
to Hemstreet (1997), hypermedia can support the complexity of PBL environments by situating a 
complex problem in a story format and making additional information about the problem readily 
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available similar to how information is built into anchored instructional materials. Hemstreet 
adds that hypermedia can also provide a database of information where more information about 
the problem than necessary is included, and the organization of the information provides no hints 
about what is important or relevant to the problem. These hypermedia problem-representation 
capabilities support the characteristics of ill-structured problems and could provide an efficient 
and effective medium to design cases for learning environments that are problem-centered. 

Additionally, hypermedia cases provide students with the capability of browsing the 
features of a case in a non-linear, multidimensional, semantic structure that mimics the brain's 
ability to store and retrieve information by referential links for quick and intuitive access (Keep 
& McLaughlin, 1995). This information seeking activity is designed to support an authentic 
process of inquiry consistent with problem-based pedagogy. It is a non-sequential activity that 
can be facilitated in a deliberate way by arranging information in a format within which 
exploration can take place. Developers, be they teachers or instructional designers, must chart 
out the knowledge students will “run into” in the course of working on a problem or case, and 
compare this to the stated goals of a course (Stepien et al., 1993). The question is however, how 
does one determine the degree and type of structure with which to arrange case information, and 
does it matter? 

Sutyak et al. (1998) argue that case structure may have a profound impact on knowledge 
transfer and students’ expectations of learning, and that case structure remains a fertile field of 
study. Additionally, Siegel et al. (2000) suggest that a high level of transfer might be attained 
through Web-based instructional design that supports case- and problem-based instruction and 
learning. Siegel et al. caution however that despite the powerful hypertext capabilities of the 
World Wide Web, many case-based designs continue to reflect a “traditional hierarchical 
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approach, modified by extended page-linking capabilities” (p. 2), inhibiting advanced knowledge 
acquisition and transfer. 

Case Structure in Web-Based Hypermedia Learning Systems 

In order to test the effects of case structure on problem-solving and knowledge transfer, 
this research project focused on hierarchical versus heterarchical Web-based hypermedia designs 
of ill-structured problems. Hierarchical hypermedia designs are linking structures (structures that 
facilitate navigation through a hypermedia system) that organize content into logical sections 
often by major topic area, resulting in several navigation levels with a top-down or tree-like 
structure (Oliver, 1996; Last, O’Donnell, & Kelly, 2001). Network-like or heterarchical 
structures are “more chaotic with random links jumping from one topic to another in a more 
referential fashion” (Oliver, 1996, p. 15). Typically, embedded links are sprawled throughout the 
content representing associations similar to knowledge representation in a memory model or 
network structure (i.e. rhizome-like). Hierarchical versus heterarchical hypermedia designs can 
also be viewed as a “depth versus breadth” topology of linking structures (Larson & Czerwinski, 
1998). For example, in a hierarchical or tree-like hypermedia structure, the deeper a student 
browses the more detail about the topic is encountered, and in order to change themes (topics) 
the student must return to the highest level (Last et. al, 2001). 

Although research indicates that a more tree-like or hierarchical structure limits 
navigational difficulties when browsing through a hypermedia learning system compared to a 
purely heterarchical structure, there is evidence that linking structures interact with learner 
characteristics (e.g. learners’ prior knowledge and cognitive styles) and with the goal of the 
learning task (e.g. studying for an exam versus exploring or searching for specific information), 
and that the design characteristics of a hypermedia learning environment can enhance or limit 
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user performance and impact the quality of independent learning (Last et. al, 2001). To further 
explore the interaction between hypermedia linking structures and student learning and 
performance, particularly in a Web-based hypermedia learning environment, this research project 
addressed the following two research hypotheses: 

1 . In a heterarchical (rhizomatic, network-like) hypermedia design of an ill-structured problem, 
significant improvement in problem-solving schemes would be observed in students who 
regard learning as an active process of constructing meaning. 

2. Students engaged in solving an ill-structured problem of heterarchical hypermedia design 
will demonstrate improved knowledge transfer compared to students engaged in solving an 
ill-structured problem of hierarchical (tree-like, topically organized) hypermedia design. 

Method of Investigation 

In order to empirically address the above research hypotheses, the first phase of this 
project is an exploratory case study to establish a set of assumptions that will guide future phases 
(Robson, 1993). This first phase serves as a diagnostic tool to study the nature of the independent 
variable (case structure) and the learning interactions, strategies, and exploratory pathways it 
generates in order to design an effective experimental study in future phases. Students’ beliefs 
and perceptions about case structure were examined by engaging them in a Web-based, PBL 
environment to solve an ill-structured problem of hierarchical and heterarchical hypermedia 
design. The following research questions were addressed in this study: 

1 . How does exploration in the heterarchical hypermedia case compare to exploration in the 
hierarchical hypermedia case? Is one more comprehensive than the other? How many links 
did students visit? Were the links visited relevant or irrelevant to the case? How much 
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additional information was explored? How much time was spent on a page/link? How many 
links were revisited to gain more clarification of the case issues? 

2. How does collaboration between group members in the heterarchical hypermedia case 
compare to collaboration in the hierarchical hypermedia case? Were students’ decisions in 
terms of how to navigate the case content group-based or individually based? Were students’ 
decisions on what actions to take while journeying through the hypermedia learning 
environment group-based or individually based? 

3. How did students in each of the groups perceive the learning task in each of these case 
designs? Was the learning task perceived as authentic, challenging, complex, ill-structured, 
meaningful, relevant, and engaging? Were these perceptions different in each group? 

4. How did students perceive the case structure in each of the groups? Did they find it difficult 
to navigate through the case? Why or why not? Did they find it difficult to locate specific 
information? 

The ill-structured problem selected for this study is an Instructional Design (ID) case 
concerning the issue of “informed consent” prior to surgery. ID cases are archetypal examples of 
ill-structured problems (Jonassen, 1997) and have been selected as study materials due to the 
expertise of the researcher in this knowledge domain. The ID problem used in this research study 
relates to pre-operative instruction in which potential gall bladder disease patients learn enough 
about the surgical procedure to give 'informed' consent. The students are responsible for 
designing instruction and assessment that legally verifies that patients were fully informed. If 
such instruction could be validated as effective, huge benefits could accrue to the medical 
profession and certainly to the insurance companies given the impending number of malpractice 
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cases filed against physicians for either misinforming or not informing patients about possible 
risks of gall bladder surgery and/or alternative procedures to correct the disease. 

The informed consent ID case described above was used in a previous research study 
(Dabbagh, et al., 2000) to argue that Instructional Design is a dynamic process of problem 
understanding and problem solution and that problem-based learning is an effective instructional 
method to teach Instructional Design because of its strategy for using problems as a stimulus for 
authentic activity and its emphasis on preparing learners to become lifelong critical thinkers. The 
case material was initially compiled in print form and presented to student groups in the 
Dabbagh et al. case study in a folder labeled by Barrows (1985) as the problem-based learning 
module (PBLM). The PBLM contained the problem statement, authentic malpractice court cases, 
samples of patients’ medical histories, articles related to the problem of informed consent, 
samples of legal informed consent forms, and legal definitions of informed consent from Black’s 
Law Dictionary. Students were also provided with related videos that demonstrated physicians 
explaining gall bladder disease and its treatment options and possible complications to patients. 

The informed consent ID case has since been converted to a Web-based hypermedia 
design reflecting the two types of hypermedia linking structures discussed above: hierarchical 
and heterarchical. The hierarchical structure consists of an unmistakable and unique entry point 
to the case which begins with a narrative of the problem scenario and branches to a Web page 
with a menu of hyperlinks to six topically labeled case themes: legal cases, definitions, article 
summaries, legal forms, patient information, and perspectives of stakeholders in the informed 
consent issue. Each of these themes is further subdivided into sub-topics menus of hyperlinks 
reflecting a tree-like navigational structure requiring students to browse deeper into the linking 
structure for more case detail, and to return to higher levels in the hierarchical organization to 
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change themes and sub-topics. Navigation in the heterarchical structure on the other hand is 
enabled through embedded textual hyperlinks that embody the natural meaning of the narrative 
articulating the case information and relevant components. The hyperlinks are sprawled 
randomly across the case content representing referential associations that resemble a network- 
like or rhizomatic structure with no discemable hierarchy or repeatable paths. Furthermore, the 
heterarchical hypermedia structure describes the case issues and events in a story-like manner 
using real world simulations of characters and role-plays deepening perhaps the authenticity of 
the problem context. The story-like nature of the heterarchical linking structure will be 
considered an attribute of the heterarchical design and its implications on student perceptions of 
the learning task will be considered only to the extent to which students perceive it as an 
important or distinctive variable of the hypermedia learning environment. 

Description of Study 

Two groups of volunteer students in their last semester of study in a graduate program in 
Instructional Design at a large comprehensive university were assigned to explore the two case 
designs described above. The researcher initially sent an email request to fifteen students asking 
them to volunteer to participate in the study. Six students responded that they would be willing to 
participate (40% response rate). The first three who responded were assigned to the heterarchical 
case design and the last three were assigned to the hierarchical case design. Students were told 
that the study will take about three hours of their time and that they would each be given a $20 
gift certificate from a major bookstore in the area to compensate for their tim e; and effort. Each 
group of students was asked to come to a computer lab on a specific date and the students were 
given instructions on how to access the case. The group assigned to the heterarchical case design 
was identified as the red group and the group assigned to the hierarchical case was identified as 
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the purple group. Students in each group were instructed to use only one computer and select an 
operator amongst them (person sitting at the keyboard) who will be in charge of accessing the 
case online and clicking the mouse. The procedure for accessing the Web-based case required 
students to enter their group name, their individual names, and the operator’s name. Then each 
group entered the URL corresponding to their case design assignment and the following 
instructions were displayed: 

This is an Instructional Design case study. Study the case and provide a one-page outline 
of your recommended solution in a Word document. While studying the case, you can 
highlight text, copy and paste portions of text, click images or links, print, etc. Every time 
you take an action, you will be asked why in a pop-up menu. Please be patient and answer 
as candidly as possible. You will not be judged or evaluated in anyway. This is a research 
experiment. When done, you will be asked to respond to a questionnaire to give us your 
suggestions. Thank you for participating in this study. Click on the arrow to enter the 
case. 

Each student group was observed by the researcher while interacting with the case and 
their journeys through the hypermedia structure were tracked to a database (through a JAVA 
program) using a pop-up menu that automatically displayed when students used any navigation 
tool to move through the pages and links of the case and/or interact with the case content (e.g. 
clicked on a hyperlink, highlighted text, copied or pasted text, printed text, used browser 
navigation tools to move backward and forward, etc.). The pop-up menu requested short answers 
to embedded elaboration-seeking questions and required students to make selections from a list 
of choices to help determine: (1) whose decision was it to initiate the interaction with case 
content (group versus individual), and (2) why was that interaction selected (relevancy to 
problem-solving process). There were five options to choose from in the second question: (a) 
very helpful for case analysis; (b) somewhat helpful; (c) not sure, but decided to explore anyway; 
(d) didn’t have any other option; (e) other reason (see figure 1 ). 
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Figure 1: Pop-up menu displayed when interaction is initiated 



A slightly different pop-up menu containing the options for question 2 was displayed 
when students exited the page that was visited, or completed the action that was selected, in 
order to determine whether students’ initial perception about the usefulness or relevancy of the 
link/action to problem-solving was confirmed. This pre-post tracking technique was necessary to 
determine what students perceived as relevant and irrelevant case information. See figure 2. 
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Figure 2: Pop-up menu displayed after visiting link or completing action 



In addition to the above information, the JAVA program recorded the URL of every page 
visited, the URL of every page in which an interaction was initiated, the start time of an 
interaction, the type of interaction, the time spent on every page/interaction, and the total session 
time. During each group session the researcher was silently observing the group processes and 
deliberations triggered while exploring the case content and taking notes. At the end of the 
session, each group submitted their outline of the case issues and recommended solution, and the 
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researcher proceeded to interview each group and explained that during the interview group 
members may jump in and answer questions at any time. The interviews were audio recorded 
and later transcribed for further analyses. The interview questions were as follows: 

1 . What did you think of the case in general? Was it easy to understand? Complex? Ill- 
defined? Well-defined? 

2. What did you think of the topic of informed consent? Was it realistic? Relevant? 
Meaningful? Was it a timely topic? Did it engage you in the case? 

3. What did you think of the structure of the case? The way it was laid out? The navigation? 
The organization of the links and resources? 

4. Was it easy to find what you were looking for? 

5. Describe the general strategy you adopted when using the system or navigating through 
the case. In other words, did you have a plan of attack? 

6. Do you think you had enough information to identify the main issues/problems to solve 
the case? 

7. Do you think all of the information presented in this case was useful? 

8. How would you change the case structure or present it differently? 

Analysis and Discussion 

This research investigation used a case study model. In addition to being an exploratory 
case study, this first phase is what Yin (1993, 1994) refers to as a positivist case study model 
where the researcher is attempting to establish the facts of the case by gathering evidence from a 
variety of resources and applying the concept of triangulation, which treats the evidence in a 
converging manner making any theory building inferences robust and viable. A qualitative 
approach was used to analyze the data. The primary data analysis techniques included content 
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analyses of the interview reports and descriptive statistics of the data collected from tracking 
students’ journeys through the two Web-based hypermedia case designs. These two different 
analyses techniques support the concept of methodological triangulation in case study research 
(Denzin, 1978; Patton, 1990). The researcher’s recorded observations and the groups’ one-page 
outlines of the case issues and recommended solutions were used as data/source triangulation 
(Denzin, 1978; Patton, 1990) to cross check the accuracy of the data collected. Methodological 
and data/source triangulation ensure the credibility of this exploratory case study. In order to 
carefully organize and interpret the data using the above techniques, a data analysis matrix (see 
table 1) was developed that maps the four research questions of this case study to corresponding 
components of the primary data sources. Table 2 shows the descriptive statistics and findings 
from the JAVA tracking program, and table 3 shows the results of the interview that was 
conducted at the end of the study. A discussion of the data analyses and implications for each 
research question is presented below. 

Research question 1 : How does exploration in the heterarchical hypermedia case compare to 
exploration in the hierarchical hypermedia case? Is one more comprehensive than the other? 

In terms of the breadth of exploration in the hypermedia cases the results showed that the 
purple group’s journeys (hierarchical) were more comprehensive overall. The purple group 
visited 54% of the links compared to 41% for the red group (heterarchical). The number of links 
revisited by the purple group was also higher than those revisited by the red group (39% 
compared to 33%). Additionally, the number of unique links visited by the purple group was 
higher than the number of unique links visited by the red group (32% compared to 27%) (see 
table 1). However, the total time spent interacting with the case content was higher for the red 
group (2 hours and 40 minutes compared to 2 hours and 7 minutes). The red group spent an 
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average of 3.04 minutes on a page compared to 1.96 minutes for the purple group. This indicates 
that the heterarchical case structure may have caused the participants to reflect more 
meaningfully on the case content, studying and discussing the information more closely in order 
to formulate a problem solution that was appropriate. The data from the interview supports this 
finding. In response to interview question 6, the participants in the red group indicated that they 
“got immersed in the materiaF’ and had to “stop and look at what we had, and go step by step 
through what we learned and what we gathered to be able to reframe the issue”. 

On the issue of usefulness and relevancy of case information to problem-solving, the 
participants in the red group indicated that some of the information was not useful but that “only 
after looking at the information we were able to determine that it was not useful” and that “even 
the unuseful information was useful in helping us come up with a conclusion”. The participants 
in the purple group pointed out that all the information was useful and that it was “good” 
information but did not elaborate on what information may have been more useful in helping 
them understand the issues of the case. The interview analyses indicated that the purple group 
seemed more focused on getting to information than on debating the issues and perspectives of 
the case content. The descriptive data indicated that both groups had the same percentage of links 
(55%) initially perceived, and later confirmed, useful, for understanding the case. However the 
red group had a higher percentage of links, 31%, initially perceived useful but later confirmed 
not useful, compared to 27% in the purple group. Additionally, the descriptive data indicated that 
the frequency reported for reason (c), “not sure but decided to explore anyway”, was 29% for the 
red group compared to 14% for the purple group. These statistics indicate that the red group was 
more willing to explore links even if their initial perception was unclear about the usefulness of 
those links in furthering their understanding of the case issues. Consequently the heterarchical 
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case structure may have caused the participants to meaningfully reflect on the usefulness of the 
case content by exploring what may have been perceived as irrelevant information in order to 
appropriately frame the issues of the case. 

The descriptive data also indicated that the purple group had a response rate of 20% 
compared to 1 1% for the red group on reason (a), “link perceived as very helpful for case 
analysis”, and a response rate of 50% compared to 29% in the red group for reason (b), “link 
perceived as somewhat helpful”. These two statistics imply that students interacting with the 
hierarchical case structure perceived more links as useful or helpful for case analysis than the 
group interacting with the heterarchical structure. However the nature of the hierarchical 
structure (i.e. the topical and layered-like organization of the links) necessitates that students 
“dig deeper” into the structure to find information, which could explain why exploration of the 
case was more comprehensive by the purple group. Interestingly though, the links that both 
groups found very helpful for case analysis (reason (a)) were very similar. Both groups found 
that malpractice cases, articles on informed consent and malpractice lawsuits, patient/physician 
interviews, and legal forms were very helpful in understanding the case issues (see table 1). This 
finding shows that both groups identified similar case information as useful but suggests possibly 
that the purple group may have been aided by the topical structure of the case design to identify 
relevant case information whereas the red group had to more carefully reason through the 
usefulness of the embedded links to identify relevant case information. 

Research Question 2: How does collaboration between group members in the heterarchical 
hypermedia case compare to collaboration in the hierarchical hypermedia case? 

In terms of collaboration between group members, the data from the computer tracking 
program showed that 97% of the decisions made by the red group were group-based compared to 
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86% in the purple group. This finding was also confirmed by the interview data. In response to 
interview question 5, the participants in the red group indicated that their strategy was 
“discussion mostly, sharing our perspectives, giving arguments for why this is important, and 
making decisions on which links to choose based on which one would help us figure out what 
informed consent was”. The purple group did not seem to be as engaged collaboratively as the 
red group. In response to question 5, the participants in the purple group indicated that they “just 
wanted a summary ... my mission was to get a summary, specific grasp on positions”. There was 
no evidence in the interview data of the purple group of discussion on how to evaluate the 
usefulness of the case content in formulating problem issues (see table 3). 

Research Question 3: How did students in each of the groups perceive the learning task in each 
of these case designs? Was the learning task perceived as authentic, challenging, complex, ill 
structured, meaningful relevant, and engaging? 

The interview data confirmed that both groups perceived the learning task (the case) as 
ill-defined, authentic, engaging and meaningful. In response to interview question 1 , the red 
group indicated that the case was “very ill-structured” and “very authentic . . . when you think of 
an instructional design project, you walk in there and don’t know the problem, don’t know the 
resources, etc.” and “in authentic cases even if you know some of those things, you don’t have 
the opportunity to get those things in the order that you want” and “also some of the links 
provided information that wasn’t helpful and that’s real life”. The purple group indicated that the 
case was “definitely ill-defined ... it’s up to the student to explore and find out what the problem 
is on their own” and “we didn’t know what the problem was even after going through some of 
the links” and “it wasn’t until after going through some links and gathering information that we 
tried to make a hypothesis as to what the problem was”. 
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In response to interview question 2 (what did you think of the topic of informed consent), 
the red group commented that the topic of informed consent was very timely and engaging 
“because people are currently having problems with HMOs (Health Maintenance Organizations) 
and they are not well informed when they go to surgery”. Similarly, the purple group commented 
that the topic was “engaging because at one point of our lives we are going to have to talk to 
doctors about possible surgeries and sign forms ... so the topic was timely and relevant”. See 
table 3 for more interview results supporting the findings for this research question. 

Research Question 4: How did students perceive the case structure in each of the groups? Did 
they find it difficult to navigate through the case? Why or why not? Did they find it difficult to 
locate specific information? 

In terms of case structure, the red group reported that the case was “structured very well” 
and suggested the addition of more pictures, audio, and video. The red group also reported that 
“there were major arteries (themes) in the case” that prompted a trail of exploration based on the 
story-like nature of the case content: ‘Tor example, on the administrative meeting page, we were 
introduced as instructional designers and then oh my god we have all these options, we could go 
to the office, the library, talk to Allen. That was a huge one, it revealed a lot of information.” In 
response to interview question 4 (was it easy to find what you were looking for), the red group 
reported that it was not easy to locate information because “we didn’t know what we were 
looking for and just like real life, this led us to have a lot of collaborative discussions ... it was a 
matter of filtering out, and that filtering out prompted collaborative discussion”. In response to 
interview question 8 (how would you change the case) the red group suggested leaving it as is 
except for the addition of more multimedia as mentioned above. 
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The purple group (hierarchical case structure) reported similar comments overall with 
some worthy-to-note variations. In response to interview question 4, the purple group reported 
that it was difficult to find information because they did not know what they were looking for. 
This was consistent with what the red group reported. In response to interview question 8, the 
participants in the purple group suggested (1) that hyperlink labels should be more 
contextualized and paraphrased to aid with the navigation; (2) that layers should be reduced; and 
(3) that information should be consolidated in the form of summaries at the first level of the 
hierarchical structure. In response to interview question 3, one participant in the purple group 
complained about the navigation: “my biggest beef was the navigation ... I felt that the 
information should have been consolidated ... it was very frustrating however the general 
structure was good”. However the other two participants in the purple group described the 
hierarchical case structure as “fine” and added that they liked how “things were spread around” 
and “how the problem scenario was the first thing to read and then there was a second layer with 
the other case perspectives, etc.”. 

These findings suggest that the heterarchical case structure may have been more 
appealing overall than the hierarchical case structure due to its embedded and more 
contextualized (story-like) linking structure and non-layered organization. More specifically, the 
finding that one student in the hierarchical case design complained about having to go through 
layers in order to get to information confirms the structural differences in the two case designs 
and suggests that top-down linking structures may prevent students from making meaningful and 
well-reasoned choices about how to explore case content. It is important to note here that the 
student from the purple group who was frustrated with the layered design of the hierarchical case 
was the operator for the group. 
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Another interesting finding was that two of the students in the purple group commented 
that they would rather go through this case on their own and come together as a group afterwards 
to discuss issues. One participant felt that “this is an exploration thing” and that “a user should be 
able to go where they want to go at any time without being restricted or guided by others”. The 
other participant added “this is a computer thing ... I don’t share computers well with others ... 
that’s why we all have our own computers”. These comments further support why collaboration 
was not as strong in this group as it was in the red group. 

The following is a summary of the results discussed above: 

• Exploration in the hierarchical case design was found to be more comprehensive in terms of 
the number of links visited/revisited, number of unique links visited, and the total number of 
interactions generated, however, the average time spent on a page in the heterarchical case 
design was one and a half times the average time spent on a page in the hierarchical case 
design. 

• Both groups identified the same links/pages as most useful with respect to relevancy to 
problem-solving suggesting that both hypermedia designs cued the learners to relevant case 
content, however, the group interacting with the heterarchical case perceived more links 
initially as useful and later determined that these links were not useful, which may imply that 
discriminating between relevant and irrelevant case information in a heterarchical design 
could be more difficult than in a hierarchical design. 

• The heterarchical case design triggered more collaboration between group members than the 
hierarchical case design. The red group’s strategy was clearly one of discussion and 
reasoning through the problem information. 
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• Perceptions of the ill-structuredness, authenticity, meaningfulness, and real-world relevance 
of the learning task were equal in both groups. 

• Perceptions of case structure in terms of organization of links, resource information, and 
navigation, were overall more positive for the heterarchical case design with the exception of 
the comment about adding more multimedia elements to the case. 

The two additional data sources (problem solutions and researcher’s notes) confirmed the 
above findings but also revealed some inconsistencies that need to be addressed in future 
research designs. In terms of the problem solutions, which were in outline form, the red group 
indicated that the key issue was largely a communication problem between doctors and patients 
regarding informed consent and that there are no clear methods to evaluate that patients fully 
understand the risks and side effects of pre-operative and operative gall bladder surgery. Their 
recommended solution was that doctors need to inform patients “completely by providing 
resources and talking about all issues” and that patients should be given specific methods to 
obtain information and gain knowledge about “how to ask doctors questions and how to research 
their medical problem”. The purple group was not as clear in their formulation of the case issues 
and problem solution. Although the group indicated that the case issue was largely one of 
communication between doctors and patients regarding informed consent, they did not suggest 
strategies on how to address this issue. Their focus was more on revising the actual forms of 
informed consent so that patients can understand better what they are consenting to. 

Both groups did not clearly identify the Instructional Design issue, which was assessment 
of patients’ knowledge about the risks and side effects of pre-operative and operative surgery. 
The case solution calls for the development of appropriate evaluation and assessment methods 
that guarantee patients’ knowledge about the risks of surgery in order to minimiz e malpractice 
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litigations against doctors and hospitals. Additionally, the case solution should include the design 
of training procedures and instructional modules that ensure that doctors and/or hospital 
personnel are effectively communicating the necessary information to patients so that patients 
can make informed decisions. The development of instructional modules or information sessions 
for patients should also be recommended. It is important to note here that this research study did 
not emphasize or focus on the problem solutions. Its primary focus was on evaluating students’ 
perceptions about hypermedia case designs of ill-structured problems and on tracking students’ 
journeys and interactions with the case in order to study the nature of the independent variable, 
case structure. Important information was revealed that would help chart the course for the next 
research study. 

The information revealed, particularly the additional time spent by the red group on the 
case, and the stronger collaboration demonstrated, justifies why their solution was perhaps more 
cogent than the other group’s solution. There was however one inconsistency noted in the 
researcher’s notes (and revealed in the tracking data as well) and that is that the purple group did 
not click on the “discussion link” which contained the actual problem question in the hierarchical 
case design. This may have impacted the problem solution of the purple group. This finding 
suggests that information in hierarchical design structures is more compartmentalized and 
prioritized according to layers and menu structures and could potentially lead to students’ 
missing out on important case information unless it is clearly delineated. To add more 
confluence to this issue, the students in the purple group (hierarchical group) did not perceive the 
organization to be more isolated or layered. They noted that other than the problem scenario, 
which was the first link they visited, the other links in the topics sub-menus seemed to be of 
equal importance. This is an inconsistency that needs to be further addressed by enhancing the 
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design of the hierarchical case to ensure that students perceive it as a layered top-down structure 
and that they do not miss out on important case information. 

Future Implications 

This exploratory case study revealed important information about Web-based hypermedia 
case structure and its impact on student learning in a problem-based learning environment. Most 
importantly, it revealed that heterarchical case designs potentially increase student collaboration 
in a Web-based hypermedia design of an ill-structured problem and engage students in thinking 
critically about case content. More research however is needed to support and solidify these 
initial findings and to address the inconsistencies noted. The heterarchical versus hierarchical 
design needs to be further studied to ensure that depth versus breadth is the delineating factor and 
that irrelevant and relevant case content can be equally integrated in both case designs. 
Additionally, different student groups (e.g. low versus high prior knowledge) should be 
subjected to hypermedia case studies using several cases with varying structure (heterarchical 
versus hierarchical) and varying problem-contexts. Case analysis and solutions need to be 
evaluated by experts in the field in terms of their viability as a proposed solution. Evaluation 
criteria in future research will include answering questions such as “Was the problem solved?”, 
“Can the learners articulate the causal relations implied by the solution?’, “How cogent was their 
argument?’, “Does it address all of the issues and constituents?’, “Did the learners effectively 
reflect their own domain knowledge?’, etc. (Jonassen, 1997, p. 86). The experts will be asked to 
rate their evaluations using such parameters on a Likert scale to quantify the problem solutions. 
Case analysis and solutions would need to be more comprehensive (i.e. not limited to a one page 
outline of case issues). Students will also be asked to respond in a written format to a list of 
questions after submitting their case analysis and solutions and will be individually interviewed 
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(instead of a group interview as was done in this study) in order to inter-rate participants’ 
responses and determine response frequencies. The questions and interviews will address 
problem design issues, problem-solving issues, and group processes. 

Since there will be multiple cases with varying structure and problem contexts in future 
research designs that identical and different groups of students will be interacting with, there will 
be two layers of data analysis, case-by-case analysis and cross-case analysis, affording a measure 
of knowledge transfer (identical groups interacting with different case-contexts). This type of 
research design is known as a multiple-case study design where the unit of analysis is a single 
case (Winegardner, 2001). In addition to being a multiple-case study design, the second phase 
will also involve a more defined experimental research design that will test the effect of case 
structure of ill-structured problems on complex problem-solving skills and knowledge transfer 
using several learning variables as predictors. This research will lead to the development of a 
design model that (1) supports the generation and Web-based hypermedia representation of ill- 
structured problem-solving tasks across multiple disciplines, and (2) ensures learning gains in 
complex problem-solving skills and knowledge transfer. Future findings will also inform, and 
possibly improve, assessment methodologies in educational research. 




26 



Web-Based Hypermedia Design 25 



References 

Barrows, H.S., & Tamblyn, R.M. (1980). Problem-based learning. New York: Springer 
Publishing Company. 

Booth-Sweeney, L. (2001). When a Butterfly Sneezes: A guide for helping kids explore 
interconnections in our world through favorite stories. Pegasus Communications. 

Brown, J.S., Collins, A., & Duguid, P. (1989). Situated Cognition and the Culture of Learning. 
Educational Researcher, 18(1), 32-42. 

Dabbagh, N., Jonassen, D.H., Yueh, H.P., Samouilova, M. (2000). Assessing a Problem-Based 
Learning Approach in an Introductory Instructional Design Course: A Case Study. 
Performance Improvement Quarterly, vl3, n3. 

Denzin, N. K. (1978). The research act: A theoretic introduction to sociological methods. New 
York, McGraw-Hill. 

Duffy, T., & Cunningham, D. (1996). Constructivism: Implications for the design and delivery 
of instruction. In D.H. Jonassen (ed.), Handbook of Research for Educational 
Communications and Technology. New York, N.Y.: Simon and Schuster Macmillan. 

Epanchin, B. & Cohicci, K. (2000). Clearinghouse for Special Education Teaching Cases. 
Federal Special Project #H29K6002, University of South Florida. [Online] Available: 
http://cases.coedu.usf.edu/default.htm 

Fitzgerald, G.E., & Semrau, L.P. (1996). Enhancing Teacher Problem-solving Skills in 

Behavioral Disorders through Multimedia Case Studies. Presented at American Educational 
Research Association 's Annual Meeting , New York, 1996. 

Gerdes, D. (1998). IMSA Center for Problem-Based Learning. Illinois Mathematics and Science 
Academy. [Online] Available: http://www.imsa.edu/team/cpbl/cpbl.html 

Grissom, S., & Koschmann, T. (1995). Hypermedia without Programming: Automatic 

Generation of Presentation Documents for Case-based Instruction. Emerging Technologies, 
Lifelong Learning, NECC 95 Proceedings. 

Hemstreet, S. (1997). Using Hypermedia to Facilitate Problem-Based Learning. Retrieved 
March, 25, 2002. http://www.edb.irtexas.edu/mmresearch/Students97/Hemstreet/index.htm 

Herreid, C.F., & Schiller, N.A. (October, 2001). National Center for Case Study Teaching in 
Science. SUNY Buffalo. [Online] Available: 
http://ublib.bufifalo.edu/libraries/projects/cases/case.html 

Jonassen, D.H. (2000). Toward a Design Theory of Problem-Solving. Educational Technology 
Research and Development 48 (4). 

Jacobson, M.J., Maouri, C., Misbra, P., & Kolar, C. (1996). Learning with Hypertext Learning 
Environments: Theory, Design, and Research. Journal of Educational Multimedia and 
Hypermedia, 5 (3/4), 239-281. 

Jacobson, M.J., & Spiro, R.J. (1995). Hypertext Learning Environments, Cognitive Flexibility, 
and the transfer of complex knowledge: an empirical investigation. 




27 



Web-Based Hypermedia Design 26 



Jonassen, D.H. (1997). Instructional design model for well-structured and ill-structured problem- 
solving learning outcomes. Educational Technology Research and Development 45 (1). 

Kagan, D. (1993). Contexts for the Use of Classroom Cases. American Educational Research 
Journal, 30(4), 703-723. 

Keep, C., & McLaughlin, T. (1995). The Rhetoric of Link Traversal, The Electronic Labyrinth. 
Hypertext Fiction Research Group. [Online] Available: 
http://jefiFerson.village.virginia.edu/elab/hfl0044.html. 

Larson, K. & Czerwinski, M. (1998). Web page design: Implications of memory, structure and 
scent for information retrieval. In Proceedings of CHI ’98, Human Factors in Computing 
Systems (LA, April 21-23, 1998), ACM press, 25-32. Copyright © 1998 by ACM, Inc. 

Lesh, R., Hoover, M., Hole, B., Kelly, A., & Post, T. (2000). Principles for Developing Thought- 
Revealing Activities for Students and Teachers. In A.E. Kelly and R.A. Lesh (Eds.) 
Handbook of Research Design in Mathematics and Science Education (pp.59 1- 645). 
Mahway, NJ: Lawrence Erlbaum Associates. 

Norman, G.R., & Schmidt, H.G. (1992). The psychological basis of problem-based learning: A 
review of the evidence. Academic Medicine, 67(9), 557-565. 

Oliver, K.M. (1996). A Critical Analysis of Hypermedia and Virtual Learning Environments. 
ERIC document: ED 412 925. 

Patton, M. Q. (1990). Qualitative Evaluation and Research Methods. Newbury Park, CA: Sage 
Publications. 

Robson, C. (1993). Real World Research: A Resource for Social Scientists and Practitioner- 
Researchers. London: Blackwell. 

Rogers, P.L., & Erickson, M. (1998). Layers of Navigation: Hypermedia Design for an ill- 
Structured Domain. ERIC document: ED 423 857. 

Schon, D.A. (1987). Educating the Reflective Practitioner. San Francisco, CA.: Jossey-Bass 
Publishers. 

Siegel, M., Derry, S., Kim, J., Steinkuehler, C., Street, J., Canty, N., Fassnacht, C., Hewson, K., 
Hmelo, C., & Spiro, R. (2000). Promoting Teachers' Flexible Use of the Learning Sciences 
through Case-Based Problem-solving on the WWW: A Theoretical Design Approach. In B. 
Fishman & S. O'Connor-Divelbiss (Eds.), Proceedings of the Fourth International 
Conference of the Learning Sciences (pp. 273-279). Mahwah, NJ: Erlbaum. 

Stepien, W., & Gallagher, S. (1993). Problem-based learning: As authentic as it gets. 

Educational Leadership, 50(7), 25-28. 

Sutyak, J.P., Lebeau, R.B., & O’Donnell, A.M. (1998). Unstructured Cases in Case-based 
Learning Benefit Students with Primary Care Career Preferences. The American Journal of 
Surgery, vol. 175, June 1998. 

The Carnegie Forum on Education and the Economy’s Task Force Report, (May 16, 1986). A 
Nation Prepared: Teachers for the 21st Century. National Center on Education and the 
Economy. 

Yin, R.K. (1993). Applications of Case Study Research. Newbury Park, CA: Sage Publications. 




28 



. Web-Based Hypermedia Design 



Yin, R.K, (1994). Case Study Research: Design and Methods, (2 nd ed.). Thousand Oaks, CA: 
Sage Publications. 

Winegardner, K.E. (accessed July, 2001). The Case Study Method of Scholarly Research. The 
Graduate School of America. [Online] Available: 
http://www.tgsa.edu/online/cybrary/case 1 .html 



Table 1: Relation of Research Questions to Primary Data Sources 



Research Questions 


Tracking Program Data Source 


Interview Questions Data Source 


(1) How does exploration in the 
heterarchical hypermedia case 
compare to exploration in the 
hierarchical hypermedia case? Is 
one more comprehensive than the 
other? 


Number of links visited, number 
or links revisited, number of 
interactions initiated, time spent 
on each interaction, type of 
interaction, total session time, 
percentage of links perceived 
useful before and after visiting. 


Question 6: Do vou think you 
had enough information to 
identify the main issues/problems 
to solve the case? 




Why was that interaction selected 
(relevancy to problem-solving 
process)? (a) Very helpful for 
case analysis? (b) Somewhat 
helpful? (c) Not sure, but decided 
to explore anyway? (d) Didn’t 
have any other option? (e) Other 
reason? 


Ouestion 7: Do vou think all of 
the information presented in this 
case was useful? 


(2) How does collaboration 
between group members in the 
heterarchical hypermedia case 
compare to collaboration in the 
hierarchical hypermedia case? 


Whose decision was it to initiate 
the interaction with case content 
(group versus individual)? 


Ouestion 5: Describe the general 
strategy you adopted when using 
the system or navigating through 
the case. In other words, did you 
have a plan of attack? 


(3) How did students in each of 
the groups perceive the learning 
task in each of these case 
designs? 




Ouestion 1 : What did vou think 
of the case in general? Was it 
easy to understand? Complex? 
Ill-defined? Well-defined? 
Ouestion 2: What did vou think 
of the topic of informed consent? 
Was it realistic? Relevant? 
Meaningful? Was it a timely 
topic? Did it engage you in the 
case? 


(4) How did students perceive the 
case structure in each of the 
groups? 




Ouestion 3: What did vou think 
of the structure of the case? The 
way it was laid out? The 
navigation? The organization of 
the links and resources? 
Ouestion 4: Was it easv to find 
what you were looking for? 
Ouestion 8: How would vou 
change the case structure or 
present it differently? 




30 



Table 2: Descriptive Statistics of Tracking Data 



Tracking Program Data 


Red Group - 
Heterarchical Case Design 


Purple Group - 
Hierarchical Case Design 


Time spent on case 


2 hours and 40 minutes 


2 hours and 7 minutes 


Number of links visited 


33 out of 81 (41%) 


79 out of 146 (54%) 


Number of revisited links 


11 out of 33 (33%) 


31 out of 79 (39%) 


Number of unique links visited 


22 out of a possible 81 (27%) 


48 out of a possible 146 (32%) 


Total number of interactions with 
the case (mouse clicks) 


34 

- 33 links 

1 text selection 


84 

79 links 

5 text selections 


Average time spent on a 
page/link 


3.04 minutes 


1 .96 minutes 


Decisions made by group 


33 out of 34 (97%) 


72 out of 84 (86%) 


Reason for interaction: 

(a) Very helpful for case 
analysis; (b) Somewhat helpful; 
(c) Not sure, but decided to 
explore anyway; (d) Didn’t have 
any other option; (e) Other 
reason. 


(a) 3 times, 1 1% 

(b) 8 times, 29% 

(c) 8 times, 29% 

(d) 4 times, 1 4% 

(e) 5 times, 1 8% 


(a) 17 times, 20% 

(b) 42 times, 50% 

(c) 12 times, 14% 

- (d) 10 times, 12% 

(e) 3 times, 4% 


Percentage of links perceived 
useful for understanding the case 
and were actually useful 


- 55% 


- 55% 


Percentage of links perceived 
useful for understanding the case 
but turned out not useful 


- 31% 


- 27% 


Case links/pages found very 
helpful 


• Cases page - contained the 
malpractice lawsuits filed by 
patients 

• Informed Consent Page - 
contained the definitions of 
informed consent 

• Research Page - contained 
links to information about the 
history of informed consent, 
articles pertaining to the 
issue of informed consent 
and malpractice lawsuits, and 
sample statements filed with 
a hospital regarding claims 

• Group was unable to access 
audio interviews 


• Cases page - contained the 
malpractice lawsuits filed by 
patients 

• Articles page - contained 
articles from scholarly 
journals about malpractice 
issues of informed consent 

• Interviews page - contained 
interviews with doctors, 
medical staff and patients 
about informed consent 

• Legal Forms - contained 
sample forms required by the 
US dept, of health and 
hospitals for informed 
consent 

• Group did not access 
discussion page link 



Table 3: Interview Results 



Interview Question 


Red Group - 

Heterarchical Case Design 


Purple Group - 
Hierarchical Case Design 


^1) What did you think 
of the case in general? 
(easy to understand, 
complex, ill-defined, 
well-defined) 


Cecilia — definitely very ill-structured 
3eth - Very authentic when you think of 
an instructional design project, you walk 
in there and don’t know the problem, 
don’t know resources, etc. 

Kathy - In ill-structured and authentic 
cases even if you know some of those 
things that you want, you don’t have the 
opportunity to get those things in the order 
you want, person’s not available, emails 
not available, etc. Also, some of the links 
provided information that wasn’t helpful 
and that’s real life. 


Dana - 1 thought it was ill defined cause at 
first we didn’t know what the problem was, 
and even after going through some of the 
links we still didn’t have an idea of what the 
problem was that we were trying to solve. It 
wasn’t until after going through some links 
and gathering information that we tried to 
make a hypothesis as to what the problem 
was. 

Claire - Agree it was ill-defined had a 
grasp of the informed consent issue from 
the initial problem statement, but I know 
there was something there that we missed I 
have a feeling there were links we 
should’ve clicked on rather than what we 
did 

Frankie — definitely ill-defined it’s up to 
the student to explore and find out what the 
problem is on their own and which is a 
good thing. 


(2) What did you think 
of the topic (informed 
consent)? 


Kathy - Very timely because in times of 
engaging society and time in the medical 
age and HMOs, people having problems 
and not being informed 
Cecilia - That applies to a real world 
example, my father goes to surgery quite a 
bit and he comes bade with options and 
we grill him about pros and cons, who 
knows what he was told, did you sign a 
form 


Frankie - Engaging to me cause at one point 
in our lives we are going to have to talk to 
doctors about our own health and possible 
surgeries we may have, so I think its 
relevant to almost anybody and easy to 
understand. I had a narrow perspective of 
what informed consent was but after going 
through the case study I can see how it can 
be applied in different situations. 

Dana - verv relevant for us cause we all 
will need to go to the doctors and sign the 
forms. It was good to see disparity of 
information that is provided to some 
patients, might be able to see that you’re 
going to a doctor who is not providing you 
with all the right information, so you need 
to find another doctor. 

Claire - Thought it was timely and relevant. 


(3) What did you think 
of the structure of the 
case? 

(well structured, ill- 
structured) 


Kathy - Think it was structured verv well. 
I liked it cause it made it an ill structured 
problem. I wouldn’t put more structure to 
it, perhaps less text, some audio, more 
multimedia, got tired of reading the text 
even though I needed to get through it, 
might’ve been other ways to present some 
of the material. I’m thinking images, 
audio clips, or movie clips might’ve made 
it more interesting. 

Cecilia -There were major arteries and 
you have a couple of options off of it so 
we follow those trails to the end and then 


Claire - Mv biggest beef was the navigation. 
We went to patient information first to get 
that perspective, question 1 then back, 
question 2 then back, felt that the 
information should’ve been consolidated, or 
a previous and next link. It was very 
frustrating. However the general structure 
was good. I’m wondering if there might’ve 
been something more obvious. I liked how 
the problem scenario at the top was the first 
thing to read, having a second layer with a 
patient, doctor, hospital administrator, 
insurance and have all of those views there 



' 


lad to go back. For example, on the 
administrative meeting page, we were 
introduced as Instructional designers and 
then oh my god we have all these options, 
we could go to the office, the library, talk 
to Allen. That was a huge one, it revealed 
lot of information. 


and have supplemental information under 
those. So could make more apples to 
oranges comparison. 

Frankie - 1 thought the structure was fine. 
Things were spread around and it wasn’t a 
linear thing, visit this first, second, third, 
which may be frustrating to some users. 
You didn’t have to give much thought to 
where things were. 

Dana - 1 agree it wasn’t linear but there was 
a theme around all the topics but didn’t 
know what it was until you went into the 
links. The only link that I didn’t know how 
}t tied was the videos. Didn’t know how the 
video of the gallbladder tied in. 


4) Was it easy to find 
what you were looking 
for? 


Kathy - Not really. we didn’t know what 
we were looking for. It wasn’t like every 
Jink gave us an “ah hah” moment. Just 
like real life, that’s what led us to have a 
lot of collaborative discussions and 
different perspective like ‘this is not 
QksfiHl -ett.was a matter of filtering out 
and that filtering out that prompted 
collaborative discussion 


frankie - Only when we knew what were 
looking for, then we knew to go back to it 
for different things, but it wasn’t at the main 
menu, only when you visited that place 
before. 

Dana - 1 agree, kind of didn’t know what 
type of information you were going to get, 
for example when you went to patient 
information you got survey questions given 
to patients. 


(5) Describe your 
general strategy while 
using the system. 


Kathy - Discussion mostly, sharing our 
perspectives, giving arguments for why 
this is important. My strategy was to 
gather as much information as possible, 
information that has something to do with 
informed consent, patient, and gallbladder. 
I would like to look at as many links as 
possible. Making decisions on which links 
to choose based on which one would help 
us figure out what informed consent was. 


Clare - Just wanted to get a summary. I 
knew there were players, my mission was to 
get a summary, specific grasp on the 
positions, we found that information was 
too specific so we went on a hunt for 
general information and that led us to legal 
information. So this was good. 


(6) Do you think you 
had enough information 
to identify the main 
issues/problems in the 
case or to solve the 
case? 


fCathv -Yes I think. I was glad that we 
stopped when we did, I had to pull myself 
back and reflect on what we looked cause 
you got so immersed in all the materials, 
bad to stop and look at what we had but I 
felt good we had a definition of what 
informed consent was. 

Cecilia - 1 think so cause towards the end 
we had got to the point where we were 
thinking is this really a problem of 
informed consent or of the doctors, ... it 
wasn’t a problem of the signing of the 
forms, it was of the doctors 
communicating to patients, and the 
patients educating themselves and being 
guided toward resources. 

Beth - At the end we were able to frame il 
better. It took to the end for us to realize 
that. We had to step back and go step by 
step what we learned, what we gathered, 
and realize that wasn’t really what the 


Frankie - I think we had enough 
information to identify the main issues but 
because it’s an ill-defined problem there is 
no 1 solution, so I don’t know if you’ll ever 
solve the problem, you may get an idea on 
perspectives, come up with guesses or 
ideas. 

Clare - Got a good grasp on the informed 
consent problem, got the sense there is a 
specific gall bladder case that had an 
informed consent issue bent around it and 
that was just illustrative of this bigger issue, 
that we didn’t get to if it does exist. 
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problem was and reframe it. It was more a 
communication issue. 




(7) Do you think the 
information was useful? 


Ceclia - some of the information was not 
useful, example the severity, we never 
went there.. .so it’s not clear. Like the 
diagnosis, it confirmed what the doctor 
said but it didn’t really help us. In the end, 
all information helped me come up with a 
conclusion, even the unuseful information 
was useful. This is the context of the 
authentic environment. Only when 
looking at the information after we had 
come up with a conclusion, we could say 
that this part wasn’t extremely important, 
but not before. 


Frankie - I think it was very useful at 
getting the different perspectives for the 
problem of informed consent specifically 
the articles, had scholarly articles, then miss 
magazine gives you down to earth 
perspective, in that sense that was good 
information. 

Dana - Information was useful, with the 
egal cases you would click on the case and 
it would bring up that information. 

Clare - We questioned the legal cases, 
woman’s son was suing the doctor cause 
medication caused pancreas disease, never 
mentioned if doctor warned her, needed 
more information about informed consent 
here. Survey questions and answers, 
particularly patient questions and answers 
seemed too detailed. A summary would’ve 
been better. 


[8) How would you 
change the case or 
present it differently to 
students? 


Beth - more multimedia, pictures, sounds, 
something to pull them in, graphics. 
Cecilia - I would leave it as it is. If its 
objective is to be problem based, authentic 
ill structured environment then leave it as 
it is ... the group was beneficial, 
collaboration was key, if I had to do this 
individually, I would’ve gone on with my 
informed consent focus and been less 
likely to look at other links that Beth 
suggested. Maybe we had arguments 
about where to go, what does that mean, 
but this keeps you on task when you’re 
with others and helps you flesh out detail 
and question things and not jump on the 
first link that you see. If no one is there to 
bold you back then you’re just going to 
elide, elide, and elide. If there is a group 
someone says something. 


Dana - Label the links differently, be more 
specific, instead of ‘questionl’, paraphrase 
the question that is asked so person knows 
where they are going and whether 
information will be useful, instead of going 
through layers and layers and finding 
information wasn’t useful. 

Clare - 1 agree, descriptions so we know 
what we’re getting into. Consolidating 
information so there are summaries. Brief 
summaries of legal cases, went to abstract, 
history... help bring that information to the 
first level so you don’t have to click back 
again. 

Frankie - 1 would prefer to go through this 
case on my own and come together 
afterwards to discuss issues. I think as we 
were going through we discussed where to 
go but I don’t think we collaborated on the 
issue in general of informed consent. In an 
exploration thing, I think a user should be 
able to go where they want to go at any 
time. 

Clare - I agree with Frankie. I don’t share 
computers well with others. Maybe it’s a 
computer thing. That’s why we all have our 
own computers. 
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